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Abstract  Holthausen,  Richard  S.;  Wisdom,  Michael  J.;  Pierce,  John;  Edwards,  Daniel  K.; 

Rowland,  Mary  M.  1994.  Using  expert  opinion  to  evaluate  a  habitat  effectiveness 
model  for  elk  in  western  Oregon  and  Washington.  Res.  Pap.  PNW-RP-479.  Port- 
land, OR:  U.S.  Department  of  Agriculture,  Forest  Service,  Pacific  Northwest  Re- 
search Station.  16  p. 

We  used  expert  opinion  to  evaluate  the  predictive  reliability  of  a  habitat  effectiveness 
model  for  elk  in  western  Oregon  and  Washington.  Twenty-five  experts  in  elk  ecology 
were  asked  to  rate  habitat  quality  for  16  example  landscapes.  Rankings  and  ratings  of 
21  experts  were  significantly  correlated  with  model  output.  Expert  opinion  and  model 
predictions  differed  for  4  of  the  1 6  landscapes.  Differences  were  most  pronounced  for 
habitats  dominated  by  large  expanses  of  either  forage  or  cover. 

Keywords:  Elk,  elk  habitat,  habitat  effectiveness,  habitat  models,  elk  management, 
model  validation,  Roosevelt  elk,  validation  research,  expert  opinion,  western  Oregon, 
western  Washington. 

Summary  Habitat  effectiveness  models  are  widely  used  by  natural  resource  agencies  to  predict 

effects  of  management  activities,  especially  timber  harvest,  on  elk  habitat.  We  evalu- 
ated such  a  model  for  elk  in  western  Oregon  and  Washington  by  using  expert  opinion. 
Experts  in  elk  ecology  rated  habitat  quality,  on  a  scale  of  0  to  1 .0,  for  16  example 
landscapes  depicted  on  computer-generated  maps.  Landscapes  represented  the  full 
range  of  habitat  conditions  likely  to  occur  within  subwatersheds  in  western  Oregon  and 
Washington.  Model  output  was  then  generated  for  the  same  landscapes. 

The  rankings  and  ratings  from  21  of  the  25  experts  were  correlated  (P<  0.05)  with  model 
predictions,  suggesting  close  agreement  between  expert  opinion  and  model  output. 
Scores  of  experts  and  the  model  differed  (P<  0.05)  for  4  of  the  16  landscapes,  however. 
Differences  were  most  pronounced  for  three  maps  that  represented  habitats  dominated 
by  large  expanses  of  either  forage  or  cover.  We  conclude  that  current  model  output  is 
a  reliable  indicator  of  expert  opinion  except  in  subwatersheds  where  the  covenforage 
ratio  is  less  than  20:80  or  greater  than  80:20.  In  these  cases,  we  recommend  using 
expert  assessments  of  habitat  effectiveness  as  a  substitute  for  model  output.  We  also 
recommend  additional  testing,  using  afield-based  standard  of  comparison,  to  refine  and 
validate  the  model. 
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Introduction  A  variety  of  habitat  models  have  been  developed  to  predict  how  wildlife  species  respond 

to  habitat  change  (for  review,  see  Morrison  and  others  1992).  How  well  these  models 
perform  depends  on  several  factors:  the  validity  of  research  data  on  which  they  are  based, 
validity  of  the  model  structure,  and  adherence  of  model  users  to  assumptions  implicit  in 
the  models  (Verner  and  others  1 986) . 

Data  used  to  develop  models  often  are  deficient,  leaving  model  predictions  biased  or  unre- 
liable (Laymon  and  Barrett  1986).  Bias  can  result  from  inadequate  sample  size,  lack  of 
replication,  confounding  variables,  poor  sampling  design,  or  inappropriate  methods  (Verner 
and  others  1986).  Thus,  model  predictions  should  be  validated  by  additional  research 
designed  to  test  and  improve  model  reliability  (Marcot  and  others  1983),  often  called  vali- 
dation research  or  validation  monitoring  (Schweitzer  and  MacNaughton  1990).  Validation 
research  is  not  always  possible;  time,  budgets,  or  other  resources  often  are  limiting.  Even 
when  possible,  it  is  rarely  accomplished  within  an  acceptable  timeframe  (Berry  1 986).  As 
an  alternative,  expert  opinion  may  be  used  instead  of  validation  research  (Montopoli  and 
Anderson  1991,  Mule  1982,  O'Neil  and  others  1988,  Schuster  and  others  1985).  In  this 
approach,  predictions  by  experts  are  compared  with  those  of  the  model  to  detect  how 
similar  they  are.  The  higher  the  positive  correlation,  the  better  the  model  is  assumed  to 
predict  a  species'  response  to  habitat  change  (O'Neil  and  others  1988). 

We  present  results  of  expert  opinion  as  a  means  to  test  the  predictive  reliability  of  a  habi- 
tat effectiveness  (HE)  model  for  Roosevelt  elk  (Cervus  elaphus  roosevelti  Merriam)  in 
western  Oregon  and  Washington  (Wisdom  and  others  1986).  Our  objectives  were  to  (1) 
determine  how  well  model  predictions  matched  those  of  experts;  (2)  identify  differences, 
if  any,  between  predictions  of  experts  and  those  of  the  model  under  an  array  of  habitat 
conditions;  and  (3)  use  results  as  a  guide  to  modify  model  structure  so  that  predictions 
better  match  those  of  experts.  These  objectives  were  based  on  feedback  from  model 
users,  who  expressed  concern  about  the  validity  of  model  predictions  for  certain  land- 
scapes. Thus,  we  assumed  that  model  output  could  be  improved  by  incorporating  the 
perceptions  of  experts,  as  done  by  O'Neil  and  others  (1988). 

Habitat  Models:  The  most  common  models  to  predict  wildlife  response  to  habitat  change  evaluate  habitat 

Concepts  and  suitability  (Berry  1986,  U.S.  Department  of  the  Interior  1981).  Habitat  models  for  elk 

Definitions  (Thomas  and  others  1 988,  Wisdom  and  others  1 986)  contain  the  same  structure  as,  and 

provide  outputs  similar  to,  models  of  habitat  suitability  but  are  referred  to  as  "models  of 
habitat  effectiveness"  (Christensen  and  others  1993,  Lyon  and  Christensen  1992).  Mod- 
els of  habitat  suitability  or  effectiveness  generally  index  potential  species  or  guild  use  of 
habitat  patches  within  a  subwatershed  or  at  another  appropriate  scale.  Predicted  use  is 
described  as  a  relative  probability  distribution,  which  is  the  relative  proportion  of  time  or 
area  that  a  habitat  patch  is  used  by  a  species  or  guild  relative  to  optimum  use  (Marcot  and 
others  1 994) .  Probability  distributions  are  scaled  from  0.0  to  1 .0,  where  1 .0  represents  1 00 
percent  probability  of  use  by  a  species  or  guild  relative  to  other  habitat  patches  having 
probabilities  less  than  100  percent.  Similarly,  0.0  is  equal  to  0  percent  probability  that  a 
particular  area  is  used  (U.S.  Department  of  the  Interior  1981).  The  sum  of  all  probabilities 
within  a  subwatershed,  weighted  by  area,  equals  the  composite,  relative  probability  of 
species  or  guild  use  for  the  entire  subwatershed  (Marcot  and  others  1994). 

Habitat  effectiveness  for  elk  is  defined  as  the  percent  of  area  or  percent  of  time  that 
habitat  is  fully  usable  by  elk  during  the  nonhunting  season  (Lyon  and  Christensen  1992). 
Models  of  habitat  effectiveness  predict  relative  use  by  elk  for  an  analysis  area  and  for  the 
composite  of  analysis  areas  within  a  subwatershed  (Thomas  and  others  1 988,  Wisdom 
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and  others  1986).  Changes  in  habitat  effectiveness  for  elk  indicate  trends  in  habitat  qual- 
ity, but  not  the  degree  of  effect  on  population  size.7  Rather,  habitat  effectiveness  models 
are  simply  a  means  for  managers  to  set  habitat  goals,  detect  trends  in  habitat,  and  assess 
compliance  in  management  of  elk  habitat  (Christensen  and  others  1 993;  see  footnote  1), 

Habitat  effectiveness  models  for  elk  are  widely  used  by  Federal  land  management  agen- 
cies in  the  Pacific  Northwest  (see  footnote  1). The  models  for  elk  have  not  been  validated,2 
although  applicable  research  is  underway  (Johnson  and  others  1991). 3  We  assume  that 
the  use  of  expert  opinion  as  presented  here  will  guide  model  refinement  until  additional 
validation  data  are  available. 

Methods  To  Gather  We  designed  1 6  maps  to  represent  the  full  range  of  variation  in  elk  habitat  that  likely  occurs 
Expert  Opinion  within  subwatersheds  of  western  Oregon  and  Washington.  Each  map  simulated  a  differ- 

ent array  of  forage  areas  (see  appendix  2  for  definitions  of  habitat  terms),  cover  types, 
and  road  densities  within  the  range  of  potential  variation  (fig.  1).  All  habitat  was  assumed 
to  be  either  a  forage  area  or  one  of  the  three  cover  types.  We  also  summarized  these  data 
in  tabular  form  (table  1)  to  complement  the  maps.  Definitions  used  to  map  and  summa- 
rize habitat  characteristics  followed  those  described  in  the  HE  model  for  elk  in  western 
Oregon  (Wisdom  and  others  1986;  appendix  2);  the  reader  is  referred  to  this  publication 
for  a  detailed  description  of  the  model,  its  assumptions,  and  calculation  of  values. 

Two  sets  of  the  1 6  maps,  each  generated  by  a  geographic  information  system  (GIS),  were 
available  for  experts  during  the  assessment.  One  set  of  maps  was  generated  at  a  scale 
of  1 :24,000  and  displayed  on  the  walls  of  the  room  in  which  the  assessment  took  place; 
the  other  set,  generated  at  a  smaller  scale  (1 :42,000),  was  provided  to  each  expert 
along  with  the  tabular  data.  This  allowed  experts  to  view  maps  and  tabular  data  clearly 
and  efficiently  during  the  entire  process. 

We  identified  25  elk  experts  from  western  Oregon  and  Washington  to  participate  in  an 
assessment  of  these  maps  and  tabular  data.  Experts  were  identified  based  on  their  expe- 
rience, knowledge,  and  professional  accomplishments  in  the  management  or  research  of 
Roosevelt  elk.  They  represented  a  broad  range  of  experience  in  management  and  research 
and  of  employers,  including  state,  Federal,  tribal,  and  private  entities. 

Before  the  assessment  began,  experts  were  asked  to  complete  a  questionnaire  about 
their  knowledge  and  experience  in  using  or  reading  about  the  elk  model  by  Wisdom  and 
others  (1 986),  and  about  their  employers  and  work  experience.  This  allowed  us  to  account 


1  Wisdom,  Michael  J.;  Rowland,  Mary  ML,  comps.  [In  preparation]. 
Procedures  for  monitoring  elk  habitat  and  populations  in  National 
Forests  of  the  Blue  Mountains.  Gen.  Tech.  Rep.  Portland,  OR:  U.S. 
Department  of  Agriculture,  Forest  Service,  Pacific  Northwest  Re- 
search Station. 

2  Rowland,  Mary  M.;  Cook,  John  G.  [In  preparation].  Validation  moni- 
toring and  research  needs  for  elk  in  the  Blue  Mountains.  Gen.  Tech. 
Rep.  Portland,  OR:  U.S.  Department  of  Agriculture,  Forest  Service, 
Pacific  Northwest  Research  Station. 

3  Cook,  J.G.;  Irwin,  LL;  Bryant,  L.D.  [and  others].  1994.  Studies  in 
elk  biology  in  northeast  Oregon:  1993  progress  report.  Unpublished 
progress  report.  [Corvallis,  OR:  National  Council  of  the  Paper  Indus- 
try for  Air  and  Stream  Improvement].  On  file  with:  NCASI,  Forestry 
and  Range  Sciences  Laboratory,  1401  Gekeler  Lane,  La  Grande 
OR  97850. 
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Figure  1 — Example  of  landscape  map  (number  12)  provided  to  experts  for  assessment  of  elk  habitat  effective- 
ness; each  landscape  encompasses  10,000  acres.  Habitat  types  are  defined  in  appendix  2. 

for  potential  variation  among  expert  assessments  due  to  differences  in  employers,  work 
experience,  or  familiarity  with  the  model. 

All  the  experts  were  given  a  set  of  written  instructions  and  a  summary  of  background 
assumptions,  in  addition  to  the  maps  and  tables,  to  guide  their  assessments.  Experts 
were  instructed  to  rank  the  maps  from  highest  (1)  to  lowest  (1 6)  habitat  quality,  and  to 
rate  each  map  for  overall  habitat  effectiveness  on  a  scale  ranging  from  0  to  1 00  percent. 
This  scale  was  the  same  as  that  produced  by  model  output  (Wisdom  and  others  1986), 
thus  facilitating  statistical  analysis  of  differences  between  expert  and  model  predictions. 


We  asked  experts  to  identify  what  habitat  factors  (for  example,  roads  or  cover;  see  appen- 
dix 1)  had  the  most  influence  on  their  assessments,  and  to  explain  why.  This  allowed  us 
to  identify  which  habitat  factors  experts  considered  most  important;  it  also  provided  addi- 
tional background  information  to  explain  potential  differences  among  expert  assessments 
and  between  expert  assessments  and  model  predictions.  Experts  also  were  asked  to 
identify  which  geographic  area  within  western  Oregon  or  western  Washington  they  were 
basing  their  assessment  on  (appendix  1).  This  enabled  us  to  test  for  differences  in  expert 
assessments  by  geographic  area. 
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Table  1 — Tabular  data  for  habitat  factors  provided  to  experts  with  maps  during  as- 
sessment of  elk  habitat  effectiveness  of  16  sample  landscapes  in  western  Oregon 
and  Washington3 


Cover  area 


Forage  Cover/forage  Road 


Map 

area 

Hiding 

Thermal 

Optimal 

Total 

edge 

density 

Thousand 

■  -  -  -/Acres  - 

feet 

Mi/mi2 

1 

10,000 

0 

0 

0 

0 

0 

1 

2 

8,100 

0 

0 

1,900 

1,900 

74 

2 

3 

3,900 

0 

0 

6,100 

6,100 

140 

4 

4 

5,600 

4,400 

0 

0 

4,400 

122 

6 

5 

2,000 

2,013 

5,987 

0 

8,000 

78 

4 

6 

2,000 

2,013 

0 

5,987 

8,000 

78 

0 

7 

2,000 

4,100 

3,900 

0 

8,000 

79 

6 

8 

3,900 

4,200 

0 

1,900 

6,100 

124 

2 

9 

2,000 

1,906 

2,094 

4,000 

8,000 

80 

1 

10 

900 

1,100 

8,000 

0 

9,100 

39 

2 

11 

1,000 

2,000 

7,000 

0 

9,000 

39 

2 

12 

1,700 

1,700 

5,000 

1,000 

8,300* 

43 

1 

13 

3,400 

0 

0 

6,600 

6,600 

91 

0 

14 

3,400 

3,400 

0 

3,200 

6,600 

86 

4 

15 

4,900 

5,100 

0 

0 

5,100 

54 

0 

16 

5,600 

4,400 

0 

0 

4,400 

86 

4 

a  Habitat  factors  are  defined  in  appendix  2. 

b  Sum  of  the  3  cover  types  does  not  equal  8,300  acres;  this  transcription  error  was  present  in  the  original  tabular 
data  given  to  experts. 


Background  assumptions  provided  to  experts  included  the  following:  (1)  that  each  map 
encompasses  a  subwatershed  of  10,000  acres  containing  year-round  (both  summer  and 
winter  range)  habitat  for  elk;  (2)  that  expert  assessment  is  based  on  current  habitat  con- 
ditions shown  on  the  maps,  and  not  on  potential  future  conditions,  given  present  habitat 
conditions;  (3)  that  elk  populations  are  hunted,  and  hunted  in  the  same  manner  across 
all  1 6  subwatersheds;  (4)  that  all  roads  are  open  to  motorized  vehicles  year-round,  are 
surfaced,  and  are  managed  similarly;  (5)  that  other  habitat  factors  not  shown  on  the 
maps,  such  as  physiography,  soils,  geology,  and  forage  quality,  are  constant  across  all 
1 6  subwatersheds  and  thus  do  not  affect  the  assessment;  (6)  that  terrain,  because  it 
was  not  considered  in  the  maps,  could  not  be  considered  a  factor  in  hiding  animals;  (7) 
that  habitat  types  (that  is,  cover  types  and  forage  areas)  are  displayed  and  summarized 
for  each  map  as  either  forage  areas,  hiding  cover,  thermal  cover,  or  optimal  cover,  using 
structural  definitions  described  for  each  type  in  Wisdom  and  others  (1 986);  and  (8)  that 
assessment  is  based  on  personal  knowledge  and  judgment  of  each  expert,  and  not  on 
the  basis  of  any  existing,  formal  habitat  models. 

Experts  made  their  assessments  by  attending  any  one  of  three  meetings  held  in  west- 
ern Oregon  and  Washington  during  June  and  July  1989.  At  each  meeting,  experts  were 
allowed  90  minutes  to  complete  their  assessments.  Each  expert  was  asked  not  to  visit 


or  talk  with  other  experts  during  the  process,  but  to  judge  the  maps  independently.  We 
provided  the  same  instructions,  maps,  data,  and  forms  (appendix  1)  to  experts  at  all  three 
meetings,  and  monitored  the  process  to  ensure  that  experts  understood  the  materials 
and  had  adequate  time  to  make  assessments. 

Methods  To  Compare  Model  predictions  of  habitat  effectiveness  were  generated  for  each  of  the  16  maps  by 
Expert  Opinion  With    using  methods  described  by  Wisdom  and  others  (1986).  Predictions  included  values  for 
Model  Predictions       three  habitat  effectiveness  variables:  (1)  sizing  and  spacing  of  forage  and  cover  areas 

(HES),  (2)  density  of  roads  open  to  motorized  vehicles  (HER),  and  (3)  cover  quality  (HEC). 
The  fourth  model  variable,  habitat  effectiveness  due  to  forage  quality  (HEF),  was  not 
computed  because  of  the  complexity  of  graphically  displaying  the  hundreds  of  possible 
combinations  of  forage  treatments  by  area  for  each  map  and  across  maps.  We  estimated 
that  asking  experts  to  evaluate  forage  conditions  would  have  more  than  doubled  the 
evaluation  period  and  increased  the  likelihood  of  experts  misinterpreting  the  instructions 
or  the  maps.  This  may  have  confounded  the  variation  of  interest  (differences  between 
the  model  and  experts'  predictions)  with  variation  due  to  misinterpretations.  Therefore,  the 
forage  variable  was  held  constant  at  0.50  (on  a  scale  of  0.0  to  1 .0)  for  all  1 6  maps.  We 
assumed  that  a  value  of  0.50  for  forage  quality  reflected  average  forage  conditions  for 
most  subwatersheds,  and  thus  was  a  realistic  rating. This  assumption  seemed  reasonable, 
based  on  our  experience  in  many  informal  applications  of  the  model  and  on  results  of  more 
formal  application  tests  across  multiple  landscapes  in  western  Oregon  (Adams  1986). 

An  overall  rating  of  habitat  effectiveness  (HESBCF)  for  each  map  was  calculated  as  the 
geometric  mean  of  the  four  HE  variables  (Wisdom  and  others  1 986).  These  habitat  rat- 
ings were  used  to  rank  the  maps  from  highest  (1)  to  lowest  (16)  habitat  quality. 

The  16  maps  were  ranked  independently  by  each  expert  from  highest  to  lowest  habitat 
quality,  based  on  the  maps  themselves  and  data  provided  in  table  1 .  Experts  were  then 
directed  to  rate  each  map,  in  terms  of  overall  habitat  effectiveness,  from  0.0  to  1 .0,  as- 
suming that  forage  quality  was  constant  across  all  maps. 

We  tested  the  null  hypotheses  that  there  was  (1)  no  correlation  (p  =  0)  and  (2)  no  differ- 
ence (HEexperts  -  HEmodel  =  0)  between  expert  opinion  and  model  predictions  through  the 
following  analyses.  Samples  were  assumed  to  be  randomly  obtained  from  a  bivariate 
normal  distribution,  and  observations  were  assumed  to  be  independent.  We  considered 
probabilities  (P)  less  than  0.05  significant. 

For  each  expert,  we  tested  for  correlation  between  map  and  model  rankings  by  using 
Spearman's  coefficient  of  rank  correlation  (Sokal  and  Rohlf  1 969,  p.  538-540).  Nonpara- 
metric  tests  for  association  are  required  when  variables  are  measured  on  an  ordinal  scale, 
such  as  the  relative  rankings  presented  here.  Likewise,  for  each  expert,  we  used  the 
parametric  test  of  correlation  analysis  (Pearson's  product-moment  correlation  coefficient 
[Sokal  and  Rohlf  1969,  p.  508-515])  to  test  for  association  between  map  and  model  rat- 
ings of  HE.  Thus,  we  had  25  tests  each  for  rankings  and  ratings. 

We  also  calculated  Pearson's  correlation  coefficient  for  the  mean  of  all  experts'  ratings 
for  each  map  and  the  model  HESRCF  score  for  the  corresponding  map.  This  provided  a 
single  rvalue  to  measure  agreement  of  experts'  ratings  with  model  output. 

For  each  map,  we  examined  differences  between  the  mean  rating  of  habitat  effectiveness 
by  all  experts  and  the  value  predicted  by  the  model,  using  a  one-sample  f-test  (Zar  1 974, 
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p.  86-90).  By  using  the  one-sample  Mest,  we  assumed  that  model  predictions  represented 
the  ideal  population,  and  that  expert  predictions  represented  samples  that  could  be  tested 
for  statistical  deviation  from  the  ideal.  Although  the  model  represented  the  ideal  popula- 
tion in  a  statistical  sense,  this  did  not  imply  that  model  predictions  represented  the  most 
correct  biological  predictions.  On  the  contrary,  our  objectives  (see  "Introduction")  assumed 
that  expert  predictions  were  biologically  more  correct  than  the  model's.  The  one-sample 
Mest  was  simply  a  statistically  appropriate  way  to  detect  such  differences. 

We  used  the  univariate  one-sample  Mest  instead  of  the  multivariate  one-sample  Hotelling's 
T2  (Hair  and  others  1 992)  to  intentionally  increase  the  probability  of  type  I  errors,  thus 
maximizing  detection  of  potential  differences  between  expert  and  model  predictions.  We 
then  could  identify  all  possible  modifications  needed  in  model  structure  to  achieve  better 
agreement  with  expert  predictions.  By  increasing  the  probability  of  type  I  errors,  we  de- 
creased the  probability  of  type  II  errors  that  can  result  from  small  sample  sizes  (such  as 
n  =  25  in  our  case).  Thus,  we  likely  increased  our  power  in  regard  to  protection  against 
type  II  errors,  but  decreased  it  for  type  I  errors  (Sokal  and  Rohlf  1 969).  This  allowed  us 
to  view  any  findings  of  no  statistical  difference  between  expert  and  model  predictions  as 
compelling  evidence  that  no  modifications  in  model  structure  were  needed  for  maps  in 
which  P  >  0.05,  a  situation  we  viewed  as  desirable  considering  our  small  sample  size  and 
the  likelihood  of  otherwise  committing  type  II  errors. 

Experts  attended  one  of  three  meetings  to  participate  in  the  evaluations,  and  chose  one 
of  four  geographic  areas  that  best  matched  the  locale  of  their  experience  in  judging  elk 
habitats.  To  determine  if  we  could  combine  data  for  analysis  from  these  geographic  areas, 
and  from  the  three  dates,  we  tested  for  differences  among  ratings  by  area  and  date  using 
one-way  analysis  of  variance  (Zar  1974,  p.  133-1 39).  We  also  examined  responses  of  ex- 
perts to  the  questionnaire  to  determine  how  many  experts  had  used  the  model  regularly. 

Finally,  we  were  interested  in  delineating  which  habitat  factors,  if  any,  were  statistically 
correlated  with  experts'  ratings  for  maps.  Thus,  we  calculated  a  separate  correlation  co- 
efficient for  each  expert's  ratings  for  the  maps  against  the  values  presented  for  the  habi- 
tat factors  (table  1).  We  performed  a  similar  correlation  analysis  of  the  individual  model 
variables  (HES,  HER,  and  HEC)  with  each  expert's  ratings. 

Results  Twenty-four  of  25  experts  (96  percent)  responded  that  they  either  had  not  used  the  model 

or  had  used  it  infrequently.  This  indicated  that  most  or  all  expert  assessments  were  likely 
independent  of  model  predictions,  an  assumption  implicit  in  our  use  of  expert  opinion  as 
a  means  of  model  evaluation.  These  results  also  indicated  that  formal  tests  of  correlation 
between  expert  familiarity  with  the  model  and  their  assessment  of  habitat  quality  were 
not  needed. 

Experts'  ratings  of  HE  differed  by  meeting  date  (P<  0.05)  for  only  three  maps,  and  by  geo- 
graphic area  for  only  one  map.  Therefore,  data  collected  from  different  meetings  and 
geographic  areas  were  combined  for  the  final  analysis;  we  also  concluded  that  further 
analysis  or  refinement  of  the  model  based  on  differences  in  expert  opinion  by  geographic 
area  were  not  warranted. 

Model  ratings  of  overall  habitat  effectiveness  for  the  16  landscapes  ranged  from  0.1 65 
(map  1)  to  0.813  (map  13)  (table  2).  Experts'  ratings  had  a  very  similar  range,  from  0.212 
to  0.829;  their  highest  and  lowest  ranking  maps  were  the  same  as  those  ranked  by  the 
model  (table  3).  Map  13,  the  highest-ranked  landscape,  had  no  roads,  a  cover :forage 
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Table  2 — Values  calculated  from  an  elk  habitat  effective- 
ness model3  for  individual  model  variables  and  overall 
index  of  habitat  effectiveness  for  each  of  16  sample 
landscapes  in  western  Oregon  and  Washington5 
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16 

.720 

.100 

.300 

.322 

a  Wisdom  and  others  1 986. 

b  Habitat  effectiveness  due  to  forage  quality  (HEF)  fixed  at  0.50  for  all  maps. 
c  Habitat  effectiveness  due  to  sizing  and  spacing  of  cover  and  forage  areas. 
d  Habitat  effectiveness  due  to  cover  quality. 
*  Habitat  effectiveness  due  to  road  density. 
'  Overall  habitat  effectiveness  index. 

ratio  near  60:40,  and  only  optimal  cover  (that  is,  no  hiding  or  thermal  cover  present).  In 
contrast,  map  1  was  the  only  one  with  no  cover  at  all.  For  individual  variables  of  the 
model,  ratings  ranged  from  0.050  to  0.952  for  sizing  and  spacing  (HES),  from  0.050  to 
1 .000  for  cover  quality  (HEC),  and  from  0.1 00  to  1 .000  for  road  density  (HER). 

Means  of  the  experts'  ratings  for  landscapes  generally  were  not  different  than  model  pre- 
dictions (table  3).  Overall  ratings  and  rankings  of  the  16  landscapes  were  correlated  (P  < 
0.05)  with  model  predictions  for  21  of  25  experts  (fig.  2).  Correlation  of  the  mean  of  all  ex- 
perts' ratings  for  each  map  with  the  model  score  also  was  significant  (r=  0.91,  P<  0.0001). 

This  general  agreement  between  experts  and  the  model  provided  rationale  for  examining 
which  individual  landscapes  were  the  source  of  the  greatest  discrepancies,  if  any,  be- 
tween expert  opinion  and  model  predictions.  One-sample  /-tests  of  differences  between 
the  mean  of  experts'  ratings  and  model  output  for  each  map  were  based  on  the  null  hypoth- 
esis that  expert  opinion  was  no  different  than  model  scores.  This  hypothesis  was  rejected 
for  only  four  landscapes,  but  landscape  9  also  was  very  close  to  the  rejection  level  of  P  = 
0.05  (table  3).  Differences  were  especially  marked  for  landscapes  2, 1 0,  and  1 1 ;  more- 
over, the  greatest  discrepancies  between  rankings  of  the  experts  versus  those  of  the 
model  were  for  landscapes  2  and  10  (table  3). 

Correlation  analysis  suggested  that,  of  the  seven  habitat  factors  for  which  data  were 
provided  to  experts  (table  1),  amount  of  optimal  cover  was  most  often  associated  with 
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Table  3 — Comparison  of  model  predictions  of  habitat  effective- 
ness for  elk  with  experts'  ratings,  for  16  sample  landscapes  in 
western  Oregon  and  Washington 
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7 
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.482 

15 

12 

9 

.425 
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16 

14 

14 

.322 
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.435 

8  Maps  ranked  from  highest  (1)  to  lowest  (16);  model  rankings  based  on  HE  ratings. 
b  Overall  habitat  effectiveness;  possible  range  of  0.0  to  1 .0. 
c  Mean  (standard  deviation);  n  =  25. 

d  Results  of  one-sample  /-test  of  model  versus  experts'  HE  ratings;  P  <  0.05  considered 
significant. 


experts'  HE  scores  (P  <  0.05  for  22  of  25  experts;  fig.  3).  In  contrast,  amounts  of  hiding 
and  thermal  cover  had  the  weakest  association  with  expert  opinion  (fig.  3).  Acres  of  total 
cover  and  forage,  along  with  road  density,  were  moderately  associated  with  experts'  HE 
ratings  (significant  correlation  for  11  of  25  experts).  The  influence  of  optimal  cover  on 
experts'  scores  also  was  reflected  in  the  correlation  between  values  of  HEC  generated  by 
the  model  and  habitat  effectiveness  ratings  for  1 7  of  the  25  experts  (fig.  3). 

Discussion  The  model  we  evaluated  (Wisdom  and  others  1986)  was  developed  from  a  combination 

of  both  empirical  data  and  expert  opinion.  At  the  time  of  its  development,  some  testing 
was  done  to  determine  if  it  was  a  reasonable  representation  of  the  opinions  of  its  devel- 
opers. That  work  represented  verification  of  the  model  formulation.  Testing  presented 
here,  which  relies  on  opinions  from  an  independent  set  of  experts,  represents  an  evalu- 
ation of  the  model  based  on  expert  opinion  for  a  full  range  of  possible  landscapes.  The 
results  demonstrate  how  well  model  predictions  are  correlated  with  expert  opinion  but  not 
whether  the  model  is  a  reliable  predictor  of  elk  habitat  quality.  Additional  research  is  needed 
to  answer  the  second  question.  The  overall  correlation  of  0.91  for  experts'  mean  ratings 
with  the  model  output  was  especially  noteworthy.  O'Neil  and  others  (1 988),  in  using  expert 
opinion  to  modify  a  HE  model,  obtained  an  rvalue  of  0.82  only  after  four  revisions  of  their 
model  for  hairy  woodpeckers  (Picoides  villosus). 

Our  results  indicated  that  model  predictions  are  a  reliable  indicator  of  expert  opinion  over 
a  broad  range  of  landscapes.  Expert  ratings  and  rankings  were  correlated  (P  <  0.05)  with 
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Figure  2 — Correlation  of  model  versus  expert  rankings  (2A;  Spearman's  correlation 
coefficient,  with  d.f.  =  16)  and  ratings  (2B;  Pearson's  correlation  coefficient,  with  d.f. 
=  1 4)  of  elk  habitat  effectiveness  on  16  sample  landscape  maps;  values  of  r  above 
the  dashed  lines  are  significant  at  P  <  0.05. 

model  output  for  21  of  25  experts.  On  a  landscape  basis,  our  null  hypothesis  that  the 
mean  of  expert  ratings  was  no  different  than  model  predictions  was  rejected  for  only 
four  landscapes.  Because  our  power  was  likely  high  for  protection  against  type  II  errors 
under  the  one-sample  f-test  (see  "Methods  to  compare  expert  opinion  with  model  pre- 
dictions"), our  finding  of  no  statistical  difference  between  expert  and  model  predictions 
for  the  majority  of  landscapes  further  verified  the  high  correlation  between  model  and 
expert  predictions. 

Differences  between  experts  and  the  model  were  greatest  for  landscapes  2, 10,  and  1 1 . 
Ratings  from  experts  were  significantly  lower  than  those  of  the  model  for  these  land- 
scapes, which  contained  a  substantial  amount  of  their  total  areas  (greater  than  80  per- 
cent) as  either  forage  or  cover.  The  only  other  test  landscape  similarly  skewed  was 
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Figure  3 — Degree  of  association  of  habitat  factors  and  elk  model  variables 
with  ratings  of  habitat  effectiveness  for  elk  by  25  experts,  using  Pearson's 
correlation  coefficient;  P  <  0.05  is  considered  significant.  Model  variables  are 
habitat  effectiveness  due  to  sizing  and  spacing  of  cover  and  forage  areas 
(HES),  due  to  cover  quality  (HEC),  and  due  to  road  density  (HER). 


landscape  12.  Explanations  given  by  the  experts  for  their  ratings  of  landscapes  2,  10,  and 
1 1  indicated  that  the  covenforage  ratio  of  these  landscapes  was  a  major  influence  in  their 
lower  ratings  of  habitat  effectiveness  versus  values  predicted  by  the  model.  Differences 
between  expert  and  model  ratings  of  these  landscapes  were  likely  influenced  by  the  rela- 
tive insensitivity  of  the  HES  variable  to  different  amounts  of  forage,  and  thus  to  covenforage 
ratios  (fig.  4).  Instead,  HES  is  sensitive  to  the  patchiness  of  forage  and  cover  areas  and 
the  percentages  of  these  areas  in  various  distance  bands  (Wisdom  and  others  1986). 
(Distance  bands  are  1 00-yard-wide  bands  away  from  cover-forage  edges). 

In  contrast,  comments  from  experts  indicated  that  their  opinions  were  sensitive  to  the 
percentage  of  area  in  forage,  particularly  when  that  percentage  was  greater  than  or  equal 
to  80  percent,  or  less  than  or  equal  to  20  percent.  This  discrepancy  between  expert  opin- 
ion and  model  prediction  over  the  HES  relationship  likely  explains  a  major  part  of  the  over- 
all discrepancy  observed  between  experts  and  the  model  for  landscapes  2,  1 0,  and  1 1 . 

Optimal  cover  was  consistently  viewed  by  experts  as  an  important  habitat  characteristic 
(fig.  3).  It  also  represented  a  key  source  of  division  for  those  whose  opinions  agreed  with 
model  output  and  those  whose  did  not.  The  three  experts  whose  scores  did  not  correlate 
with  amount  of  optimal  cover  also  were  three  of  the  four  whose  overall  opinions  did  not 
match  the  model  (figs.  2  and  3).  Other  habitat  factors,  such  as  roads  and  total  forage 
acreage,  were  strongly  associated  with  these  experts'  scores. 

Conclusions  Habitat  effectiveness  predicted  by  the  model  agreed  closely  with  expert  opinion  except 

when  landscapes  contained  more  than  80  percent  of  the  area  in  either  forage  or  cover. 
Model  predictions  of  habitat  effectiveness  were  higher  than  those  of  the  experts  in  those 
cases.  The  insensitivity  of  HES  to  extreme  values  for  forage  and  cover  may  be  respon- 
sible for  this  difference,  as  may  be  the  model's  structure.  The  use  of  the  geometric  mean 
in  combining  scores  for  individual  model  variables  allows  partial  compensation  for  low 
scores  of  one  variable  with  higher  scores  for  other  variables. 
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Figure  4 — Relation  between  size  of  forage  area  and  the  model  variable,  habitat  effec- 
tiveness due  to  sizing  and  spacing  of  cover  and  forage  areas  (HES). 

We  recommend  that  model  structure  be  modified  to  better  match  expert  opinion  when 
covenforage  ratios  are  less  than  20:80  or  greater  than  80:20.  Specifically,  we  recom- 
mend that  these  values  be  set  as  thresholds.  When  such  thresholds  are  exceeded, 
overall  habitat  effectiveness  (HESRCF)  should  be  lowered  to  0.4.  This  value  approximates 
expert  opinion  (based  on  results  from  maps  2, 10,  and  1 1 ;  table  3).  If,  however,  HESRCF 
predicted  by  the  model  is  less  than  0.4  in  these  cases,  we  recommend  that  model  pre- 
dictions be  used  in  unmodified  form  (see  results  for  map  1 ;  table  3). 

These  recommendations  should  be  taken  cautiously  owing  to  the  small  sample  size  on 
which  they  are  based.  These  recommendations  do  contradict  results  from  map  1 2,  where 
covenforage  ratios  were  greater  than  80:20,  but  the  model  and  experts  agreed  with  a  HE 
value  of  about  0.57  (table  3).  The  low  road  density  for  this  landscape  may  have  been 
responsible  for  this  agreement.  Until  validation  research  of  all  model  variables  and  their 
interactions  is  undertaken,  however,  these  recommendations  appear  reasonable  for  most 
cases.  They  are  based  on  the  assumption  that  expert  opinion  can  serve  as  an  interim 
guide  for  modifying  model  output  when  the  two  methods  differ. 

We  also  recommend  that  model  users  calculate  HEF  based  on  field  records  of  forage 
treatments,  rather  than  holding  this  value  constant  at  0.50.  We  held  this  variable  constant 
only  because  of  the  difficulties  in  displaying  the  myriad  of  treatment  combinations  to  the 
experts.  This  does  not  imply  that  HEF  is  unimportant  or  should  be  held  constant  during 
actual  evaluations  of  HE. 


Differences  between  experts'  and  the  model's  sensitivity  to  covenforage  relations  can 
be  established  from  several  different  paths  of  inference,  including  (1)  examination  of  the 
statistical  differences  in  ratings  for  maps  2, 1 0,  and  1 1 ;  (2)  the  apparent  differences  in 
ranking  of  maps  2  and  10;  and  (3)  experts'  statements  about  factors  that  influenced  their 
judgment  of  the  three  maps.  The  inference  would  have  been  much  weaker  if  it  had  been 
based  solely  on  ratings.  Ratings  imply  some  calibration  of  the  model  to  absolute  values 
of  habitat  effectiveness,  but  such  calibration  is  problematic.  The  correlation  established 
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in  this  study  between  model  and  expert  ratings  (fig.  2B)  may  have  been  influenced  by 
the  decision  to  have  HEF  equal  0.50.  A  different  value  for  HEF,  or  holding  a  different  vari- 
able constant,  could  have  significantly  altered  this  correlation.  Thus,  conclusions  that  can 
be  based  on  rankings  are  likely  less  biased  than  those  based  on  ratings.  Interpretation 
of  model  results  will  be  more  valid  if  ratings  are  viewed  as  relative  rather  than  absolute. 
Also,  statistically  significant  differences  between  ratings  of  experts  and  the  model  do  not 
necessarily  imply  that  biological  differences  exist. 

This  work,  and  previous  work  by  Montopoli  and  Anderson  (1 991)  and  O'Neil  and  others 
(1 988) ,  demonstrate  the  usefulness  of  professional  opinion  to  validate  habitat  models.  This 
procedure  may  be  especially  effective  when  models  are  controversial  and  model  users 
question  a  model's  utility.  During  our  study,  many  of  the  experts  became  more  comfortable 
with  the  model  after  participating  in  a  structured  comparison  of  their  opinion  with  model 
results.  These  experts  may  influence  the  opinions  of  many  other  potential  model  users. 
This  work  also  has  provided  a  useful  basis  for  making  relatively  minor  modifications  to 
the  model. 

Despite  the  usefulness  of  expert  opinion,  it  should  not  be  considered  a  substitute  for  vali- 
dation based  on  field  research.  Expert  opinion  used  to  evaluate  the  model  may  not  be 
any  more  robust  than  the  opinion  used  to  create  the  model,  and  may  contain  significant 
biases.  Validation  based  on  field  research  must  be  used  to  detect  biases.  Such  research 
may  not  be  completed,  or  even  started,  in  the  near  future,  however.  Evaluation  by  expert 
opinion  serves  as  a  useful  interim  step  for  those  obligated  to  conduct  habitat  assessments 
using  HE  models  before  full  validation  can  be  completed. 
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Appendix  2:  Habitat  terms  used  by  experts  in  evaluations  of  habitat  effectiveness. 

Definitions 

Cover/forage  edge:  the  linear  distance  of  the  boundaries  where  forage  and  cover  areas 
abut. 

Forage  areas:  vegetated  areas  with  less  than  60  percent  combined  canopy  closure  of 
trees  and  tall  shrubs  (only  trees  and  shrubs  over  7  feet  in  height  are  considered  in  this 
definition). 

Hiding  cover:  any  vegetation  capable  of  hiding  a  standing  adult  deer  or  elk  at  200  feet 
or  less — generally  all  stands  in  western  Oregon  and  Washington  not  qualifying  as  for- 
age areas  or  optimal  or  thermal  cover. 

Optimal  cover:  forest  stands  with  dominant  trees  averaging  21  inches  in  diameter  at 
breast  height  (d.b.h.)  or  larger,  70  percent  crown  closure  or  higher,  and  in  the  large 
sawtimber  or  old-growth  stand  condition. 

Road  density:  number  of  miles  of  road  per  square  mile  of  habitat;  for  this  exercise,  roads 
are  assumed  to  be  surfaced,  managed  similarly,  and  open  to  motorized  vehicles  year- 
round. 

Thermal  cover:  stands  at  least  40  feet  in  height  with  tree  crown  closure  of  at  least  70 
percent — generally  achieved  in  closed  sapling/pole  stands  or  older  stands  not  qualify- 
ing as  optimal  cover. 
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Holthausen,  Richard  S.;  Wisdom,  Michael  J.;  Pierce,  John;  Edwards,  Daniel 
K.;  Rowland,  Mary  M.  1994.  Using  expert  opinion  to  evaluate  a  habitat  effec- 
tiveness model  for  elk  in  western  Oregon  and  Washington.  Res.  Pap.  PNW-RP- 
479.  Portland,  OR:  U.S.  Department  of  Agriculture,  Forest  Service,  Pacific 
Northwest  Research  Station.  16  p. 

We  used  expert  opinion  to  evaluate  the  predictive  reliability  of  a  habitat  effective- 
ness model  for  elk  in  western  Oregon  and  Washington.  Twenty-five  experts  in  elk 
ecology  were  asked  to  rate  habitat  quality  for  16  example  landscapes.  Rankings 
and  ratings  of  21  experts  were  significantly  correlated  with  model  output.  Expert 
opinion  and  model  predictions  differed  for  4  of  the  16  landscapes.  Differences  were 
most  pronounced  for  habitats  dominated  by  large  expanses  of  either  forage  or 
cover. 

Keywords:  Elk,  elk  habitat,  habitat  effectiveness,  habitat  models,  elk  management, 
model  validation,  Roosevelt  elk,  validation  research,  expert  opinion,  western 
Oregon,  western  Washington. 
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